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INTRODUCTION 

Known as “the Nation’s Report Card,” 
the National Assessment of Educational 
Progress (NAEP) is the U. S. Govern- 
ment assessment used since the late 
1960s to measure student achievement 
in many subject areas, including math- 
ematics, reading, science, and United 
States history. Different subject areas are 
assessed each year and data have been 
available for all states plus the District 
of Columbia and Department of Defense 
Schools since the mid 1990s. The 2013 
NAEP assessed mathematics and read- 
ing performance and results were re- 
leased in November. Comments made 
in the Indianapolis Star were typical: 
“Hoosier students’ gains on a national 
report card ranked among the top four 
states in math and reading” (Wang, 
2013). Indeed, Indiana ranked in the top 
four for increase in average scale scores 
from 2011 to 2013 for grade 4 math- 
ematics (with Tennessee, the District 
of Columbia, and Arizona) and grade 4 
reading (with Tennessee, the District of 
Columbia, and Minnesota). The increase 
in grade 8 scores was less impressive but 
Indiana’s improvement was still strong, 
ranking in the top 8 for mathematics and 
the top 15 for reading. 

Indiana has a history of doing reason- 
ably well on national and international 
assessments. Chein, Spradlin, and 
Plucker (2007) found that Indiana stu- 
dents outperformed the nation on aver- 
age in grades 4 and 8 mathematics, and 
also had the highest or second highest 
average scale scores for 2000, 2003, 
and 2005 when compared to neighbor- 
ing states. In addition, Rutkowski, Wild, 


and Rutkowski (2013) reported strong 
performance on the 2011 Trends in Inter- 
national Mathematics and Science Study 
(TIMSS) mathematics assessment: Indi- 
ana’s grade 4 and 8 students performed 
better than the United States as a whole 
and “compared favorably to the Top 10 
performing countries” (p. 7). 

U.S. Secretary of Education Arnie Dun- 
can credited Indiana’s “work to raise 
standards and target teacher effective- 
ness” (Wang, 2013) for the strong 2013 
NAEP results. Former State Superinten- 
dent of Public Instruction Tony Bennett 
asserted that “the policy framework we 
put in place afforded schools the op- 
portunity to expect more of children” 
(Elliott, 2013), while Teresa Meredith, 
president of the Indiana State Teachers 
Association, gave credit to Indiana’s 
state standards movement in the early 
2000s. Education blogger Steve Hin- 
nefeld (2013) wondered if the imple- 
mentation of the grade 3 IREAD test in 
the 2011-2012 school year — and, in par- 
ticular, the many students who did not 
pass and were held back in third grade — 
may have had an effect. In this report, 
we provide details of the 2013 Indiana 
NAEP results followed by commen- 
tary on the extent to which state-level 
policies and priorities had an impact on 
those results. 


USING NAEP TO COMPARE 
STATES 

Before detailing the performance of In- 
diana students on NAEP, it is important 
to note that the number of students as- 
sessed in each state is relatively small. In 
Indiana, NAEP tested about 3000 grade 
4 students and about 2600 grade 8 stu- 


dents in each of the two content areas. 
The students tested are representative of 
the gender, racial, and socioeconomic 
demographics in Indiana and thus while 
the sample sizes are small, they are simi- 
lar to sample sizes in previous years and 
adequate for providing an overall sense 
of the performance of Indiana students. 
However, the small state samples lead to 
a relatively large margin of error when 
comparing states. It is also important 
to note that all state-level data reported 
by NAEP are for public school students 
only. NAEP collects data on students in 
private schools for national results but 
there are not enough private school stu- 
dents in some states to draw conclusions 
and thus all state-level data, including 
the NAEP results reported for Indiana, 


are for public school students only. To 
allow for comparison of Indiana data to 
national results, all national results dis- 
cussed in this brief are for public school 
students. 


2013 MATHEMATICS RESULTS 
FOR INDIANA 

In 2013, Indiana’s average scale score 
ranked 4th among the 50 states plus DC 
and Department of Defense Schools for 
grade 4 mathematics. Given the impreci- 
sion in state scores, however, the Indiana 
score was significantly higher than only 
38 states. In grade 8 mathematics, Indiana 
tied for 16th place but was significantly 
lower than 6 states and significantly 


higher than 25 states. In other words, all 
that can be said is that Indiana’s average 
mathematics scores were somewhere be- 
tween 4 th and 14 th at grade 4 and between 
7 th and 27 th at grade 8. Table 1 shows the 
states and jurisdictions higher, equal to, 
and lower than Indiana in mathematics. 


2013 READING RESULTS FOR 
INDIANA 

Although Indiana’s average scale score 
was tied for 1 5 th in reading at grade 4 and 
tied for 26th at grade 8, the same ambi- 
guity in rankings that appeared in math- 
ematics occurs in reading. Thus, Indiana 
4th graders were somewhere between 
7th and 29th and eighth graders were 


TABLE 1. Indiana’s Ranking Relative to Other States and Jurisdictions on NAEP 2013 Mathematics 


States significantly higher 
No significant difference 
States significantly lower 


States significantly higher 
No significant difference 

States significantly lower 


Number 


States* 


Grade 4 

3 MA, MN, NH 

10 VT, CO, NJ, WY, KS, ND, OH, VA, WA, MD 

38 IA, ME, DoDEA, NC, Wl, MT, PA, CT, DE, HI, NE, UT, FL, TX, ID, KY, Rl, SD, 

AZ, AR, GA, MO, NY, OR, TN, IL, OK, Ml, SC, WV, AK, NV, CA, AL, NM, LA, 

MS, DC 

Grade 8 

6 MA, NH, NJ, MN, VT, ND 

20 CO, DoDEA, KS, OH, PA, WA, ME, MT, Wl, TX, VA, WY, MD, SD, ID, NC, CT, IL, 
IA, NE 

25 OR, Rl, UT, MO, AK, DE, NY, FL, HI, KY, AZ, Ml, SC, GA, AR, NV, TN, CA, OK, 
WV, LA, NM, MS, AL, DC 


includes change in scores for the District of Columbia and the Department of Defense Education Activity. 


TABLE 2. Indiana’s Ranking Relative to Other States and Jurisdictions on NAEP 2013 Reading 


Number 


States* 


States significantly higher 
No significant difference 

States significantly lower 


States significantly higher 
No significant difference 

States significantly lower 


Grade 4 

6 DoDEA, MD, MA, NH, CT, VT 

22 NJ, VA, CO, FL, MN, DE, PA, WY, ME, WA, IA, KY, NY, ND, OH, KS, MT, NE, Rl, 
UT, MO, NC, 

23 GA, Wl, TN, AL, AR, ID, IL, OR, SD, Ml, OK, TX, HI, WV, NV, SC, AZ, CA, LA, 
AK, MS, DC, NM 

Grade 8 

13 DoDEA, MA, NJ, CT, MD, NH, VT, MT, PA, WA, CO, MN, WY 
23 ID, KY, UT, IA, ME, NE, OH, ND, OR, SD, VA, Wl, IL, KS, MO, Rl, DE, FL, Ml, 
NY, GA, NC, TN 

15 TX, AR, CA, NV, OK, AK, SC, AZ, HI, AL, LA, WV, NM, MS, DC 


includes change in scores for the District of Columbia and the Department of Defense Education Activity. 
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FIGURE 1. NAEP Mathematics Average Scale Scores for Indiana and the Nation, 1990-2013 
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Note: The difference between Indiana and the nation is statistically significant (p < .05) for both grades for all years except 
grade 4 in 1992 and grade 8 in 2011. 


TABLE 3. NAEP Achievement Level Descriptions 


Level Description 

Basic Partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade. 

Proficient Solid academic performance for each grade assessed. Students reaching this level have demonstrated 

competency over challenging subject matter, including subject-matter knowledge, application of such knowledge 
to real-world situations, and analytical skills appropriate to the subject matter. 

Advanced Superior performance. 


between 14 th and 37 th . Table 2 shows the 
states and jurisdictions higher than, equal 
to, and lower than Indiana for reading. 


GROWTH OVER TIME IN 
MATHEMATICS 

Most NAEP items are used for several 
years making it possible to assess chang- 
es in performance over time. Figure 1 
shows the average scale scores on NAEP 
for Indiana and the nation’s fourth and 
eighth graders from 1990 to 2013. From 
1996 to 2000, NAEP transitioned from 
an assessment that did not allow accom- 
modations for students with disabilities to 
one that does. In some cases, changes of 
this sort can mean that comparisons be- 


fore and after are not valid. For NAEP, 
however, the change in students’ aver- 
age scores was small, no more than two 
points for the years when students took 
both versions. Given this small differ- 
ence, considering trends from 1990 to 
2013 can be justified (Kloosterman & 
Walcott, 2007). 

As can be seen in Figure 1, except for 
1992, Indiana’s grade 4 students have 
significantly outperformed the nation 
on mathematics. In addition, Indiana’s 
grade 8 mathematics scores have been 
significantly above the national aver- 
age in every testing year except 2011. 
Of equal importance is the fact that, like 
the nation as a whole, Indiana’s scores 
have improved steadily since 1990. 
Specifically, Indiana’s grade 4 average 


scale scores improved from 22 1 in 1 992 
to 249 in 2013. For grade 8, the scores 
improved from 267 in 1990 to 288 in 
2013. Kloosterman and Walcott (2007) 
estimated that an 11- to 12-point gain is 
approximately one grade level of growth 
for fourth graders and an 8- to 10-point 
gain is approximately one grade level of 
growth for eighth graders. Using this es- 
timate, Indiana’s 2013 fourth graders are 
at least 2 grade levels above 1992 fourth 
graders. Similarly, Indiana’s 2013 eighth 
graders performed at least 2 grade lev- 
els above their 1990 counterparts. These 
gains, which are similar to gains for the 
nation as a whole, are substantial and far 
more than gains in any other content area 
assessed by NAEP. 

In addition to reporting NAEP results 
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FIGURE 2. Percent of Indiana Students at Each NAEP Achievement-Level for Mathematics 
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by overall scale scores, NAEP reports 
scores by the percentage of students at 
each of four achievement levels: below 
basic, basic, proficient, and advanced 
(Table 3). Although it is often assumed 
that these levels are set by official pol- 
icy, they are only being used on a trial 
basis meaning that “achievement levels 
should continue to be interpreted and 
used with caution” (National Center for 
Educational Statistics, 2012). The levels 
are related to scale scores so while they 
do not provide any different informa- 
tion than scale scores, they do provide 
a more focused way of interpreting the 
data. Many analysts have found prob- 
lems with the achievement levels (see, 
for example, Harvey, 2011), but even if 
the criticisms of the particular cut-scores 


used for the achievement levels are val- 
id, changes in the percent of students at 
the higher levels is evidence of improve- 
ment on the assessments. 

As Figure 2 shows, the proportion of In- 
diana fourth graders scoring at the profi- 
cient or higher level in mathematics has 
increased from 16% in 1992 to 54% in 
2013. Similarly, the proportion of fourth 
graders scoring at the advanced level has 
increased from 1% in 1992 to 10% in 
2013. The shift for eighth graders is less 
pronounced, but still large and statisti- 
cally significant: from 17% proficient or 
better in 1990 to 38% proficient or better 
in 2013. The percent of students at the 
advanced level increased from 3 % in 
1990 to 10% in 2013. 


Some have criticized the cutoffs for the 
NAEP proficient level as too high (e.g., 
Rothstein, Jacobson, & Wilder, 2006) al- 
though Chester Finn (2008), chair of the 
National Assessment Governing Board 
(NAGB) when the NAEP assessment 
levels were established, described profi- 
cient as “the ‘central level,’ the one that 
all students ought to attain and the prop- 
er benchmark for American education” 
(p. 180). Diane Ravitch (2013), a former 
NAGB member herself, described profi- 
cient differently: “a solid A and not less 
than a B+” (p. 47). Furthermore, she de- 
scribed basic as “probably a B or C” (p. 
47). By Ravitch’s standard, basic may 
be a better indication of whether students 
are performing at grade level. The de- 
crease in the percentage of grade 4 stu- 
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FIGURE 3. NAEP Reading Average Scale Scores for Indiana and the Nation, 2002-2013 
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Note: The difference between Indiana and the nation is statistically significant (p < .05) in 1992, 1994, 2002, 2003, 2007, 2009, 
and 2013 for grade 4 and 2003, 2007, and 2009 for grade 8. 


FIGURE 4. Percent of Indiana Students at Each NAEP Achievement-Level for Reading 
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dents below basic has been substantial, 
from 40% in 1992 to just 10% in 2013. 
At grade 8, the change has been less, but 
still large with 44% below basic in 1 990 
and only 23% below basic in 2013. Most 
of the improvement, however, was in the 
1990s - the percent of students scoring 
below basic in 2013, is not substantially 
different than 2000. 


GROWTH OVER TIME IN 
READING 

Figure 3 shows the Indiana and na- 
tional average scale scores in reading 
from 1992 to 2013. Like mathematics, 
both the grade 4 and grade 8 results in- 
clude a break in the trend line because 
of a change in accommodation policies. 
Grade 4 also includes a gap in the Indi- 
ana results because state-level data were 
not collected in 1998 or 2000. For grade 
8, state-level data were not collected un- 
til 2002. 

For grade 4, Indiana’s reading average 
scale scores showed a statistically signif- 
icant change from 221 in 1992 to 225 in 
2013. The practical significance of such 
a small change, however, seems slight. 
For grade 8, the change, from 265 in 
2002 to 267 in 2013, was not statistically 
significant. For both grades, the trends 
for Indiana are similar to the trends for 
the nation as a whole. 

NAEP reports reading results using the 


same achievement levels as mathemat- 
ics: below basic, basic, proficient, and 
advanced (Table 3). As would be expect- 
ed by looking at the minimal change in 
scale scores, the percentage of U. S. stu- 
dents performing at proficient or above 
has changed only modestly over the last 
20 years. At grade 4, the percentage scor- 
ing at proficient or advanced increased 
from 32% in 1992 to 38% in 2013 and 
at grade 8, the increase was from 32% 
in 2002 to 35% in 2013 (Figure 4). With 
respect to Indiana, more grade 4 students 
were at basic or above in reading (68%) 
as compared to mathematics (60%) in 
1992 but with greater improvements in 
mathematics there is now a higher pro- 
portion of grade 4 students in Indiana 
and nationally below basic in reading as 
compared to mathematics. At grade 8, 
the proportion of students at or above ba- 
sic in mathematics and reading was quite 
different in 1990 (56% vs. 77%) but is 
now almost the same (77% vs. 79%). 


CHANGE IN SCORES FROM 
2011 TO 2013 

Although the overall trend for NAEP 
average scale reading scores has been 
relatively flat, there was a significant 
jump in grade 4 Indiana scores from 22 1 
in 2011 to 225 in 2013. In mathemat- 
ics, the overall trend has featured steady 
increases, but the change from 2011 to 
2013 in grade 4 (244 to 249) was larger 


than usual. Much of the reporting on 
this change has focused on the fact that 
these changes were the 4th largest of any 
state (see, for example, Wang, 2013). 
As with average scale scores, however, 
not all differences in ranking are statisti- 
cally significant. In fact, while no state 
or jurisdiction had changes in scores 
significantly greater, Indiana’s grade 4 
improvements were statistically higher 
than only about half of the states and 
jurisdictions (Table 4). At grade 8, In- 
diana’s improvement was statistically 
equivalent to most other states (Table 4). 


RESULTS FOR DEMOGRAPH- 
IC SUBGROUPS 

In addition to overall results, NAEP re- 
ports results by a variety of demographic 
variables including gender, race-ethnic- 
ity, and free or reduced-price lunch eli- 
gibility. When Indiana data are broken 
down by subgroup, the trends are quite 
similar to the trends for the nation as a 
whole. With respect to gender, Indiana’s 
boys scored enough higher to be statisti- 
cally better than girls in mathematics for 
some 1990s test administrations (1996 
for grade 4; 1992 and 1994 for grade 8) 
but there have been no statistically sig- 
nificant differences since. At the nation- 
al level, the differences tend to be one 
or two scale points, but with the large 
sample size, these differences are usual- 
ly statistically significant. Nationally in 


TABLE 4. Indiana’s Change in NAEP Average Scale Scores Relative to Other States and Jurisdictions from 2011 to 2013. 


Number of States* 


Mathematics Reading 



Grade 4 

Grade 8 

Grade 4 

Grade 8 

States with significantly more improvement 

0 

0 

0 

0 

No significant difference in improvement 

25 

43 

28 

49 

States with significantly less improvement 

26 

8 

23 

2 


includes change in scores for the District of Columbia and the Department of Defense Education Activity. 


2013 NAEP — 6 


FIGURE 5. 2013 Indiana and National Public NAEP Mathematics Average Scale Scores by Race/ethnicity 
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FIGURE 6. 2013 Indiana and National Public NAEP Reading Average Scale Scores by Race/ethnicity 
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FIGURE 7. 2013 Indiana and National Public NAEP Mathematics Average Scale Scores by Free- or Reduced-lunch 
Eligibility 
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FIGURE 8. 2013 Indiana and National Public NAEP Reading Average Scale Scores by Free- or Reduced-lunch 
Eligibility 
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reading, girls score about 7 points better 
than boys at grade 4 and about 10 points 
better than boys at grade 8 depending on 
the year. Because of the small Indiana 
sample size, there tends to be more vari- 
ation in the gap from year to year, but the 
overall Indiana gap is quite similar to the 
national gap. 

When data are analyzed by the NAEP 
race-ethnicity variable, differences by 
subgroup are far more dramatic than 
differences by gender. Figures 5 and 
6 show the 2013 Indiana and national 
mathematics and reading average scale 
scores broken down by the three of the 
five race-ethnicity categories used by 
NAEP: white, black, and Hispanic. The 
other two categories, Asian/Pacific Is- 
lander and American Indiana/ Alaska 
Native were not included because, for 
nearly all test administrations, the num- 
ber of Indiana students in these catego- 
ries was too small to draw any conclu- 
sions. As can be seen, regardless of 
grade level or subject area, the highest 
scoring of these subgroups is white stu- 
dents. Hispanic students’ scores were 
substantially below white students, and 
black students were still lower. As was 
the case with gender, Indiana results 
are similar to those for the nation as a 
whole. It is important to remember that 
the figures represent averages for each 
subgroup and that there are many high 
and low performing students in each cat- 
egory. That being said, these are large 
differences. Although the NAEP scaling 
system is not designed to compare grade 
4 to grade 8 performance, the difference 
between the fourth and eighth grade 
2013 average scale score was 43 points 
in mathematics (Figure 1) and 45 points 
in reading (Figure 3). The differences 
between various groups is always less 
than the difference between grade 4 and 
8 but not always that much. The extent 
to which closing achievement gaps can 
be documented depends on the data and 
methodology used (Brown Center, 2012; 
Rowan, Hall, & Haycock, 2010) but the 
gaps are so large that a prolonged effort 
will be necessary if there is any hope of 
equalizing educational outcomes based 
on racial or ethnic status. 


Related to the issue of substantial 
achievement gaps based on race and eth- 
nicity is the issue of achievement gaps 
based on income. Figures 7 and 8 shows 
the 2013 average scale scores for stu- 
dents who were and were not eligible for 
free- or reduced-price lunch. As can be 
seen in the figures, (a) the gaps are simi- 
lar in magnitude to race-ethnicity gaps 
and (b) while Indiana students scored 
a bit higher than the national sample in 
2013, the gaps based on free or reduced 
price lunch eligibility are similar for 
Indiana and the nation as a whole. Of 
equal importance is that with the high 
correlation between socio-economic sta- 
tus and race-ethnicity, it has been argued 
that the racial and ethnic gaps are related 
to SES gaps and thus it will be difficult 
to close such gaps without addressing is- 
sues of economic disparity (Eubienski & 
Crockett, 2007). 


INDIANA NAEP RESULTS AND 
STATE-LEVEL POLICY 

It is always tempting to look at changes 
in scores on tests like NAEP as evidence 
of the effects of recently implemented 
policies, but education is a complex 
endeavor and such causal links cannot 
be definitively made. The influence of 
state standards may be a factor, as might 
changes in state education policy. Still, 
the implementation of IRE AD- 3 and the 
changes in grade 3 retention rates are 
very likely to be the major reason why 
grade 4 Indiana students scored so much 
higher in mathematics and reading in 
2013 than their counterparts in 201 1 . 

In 20 1 1 -20 12, 1 5 . 1 % of the third graders 
tested did not pass IREAD-3. Although 
there are no statewide data on the exact 
percentage of students held back, we 
know that 6.9% of the 2011-2012 third 
graders received exemptions allow- 
ing promotion to fourth grade (Indiana 
Department of Education, n.d.). This 
suggests that up to 8% of students that 
would have been promoted to grade 4 in 
2012-2013 remained in grade 3. Warren 
and Saliba (2012) estimated that Indi- 
ana’s retention rate for grade 3 was 2% 


or less from 2002 through 2009. Thus, 
relative to prior years, it is reasonable to 
assume that many of the weakest read- 
ers - comprising what would have been 
about 6 % of the grade 4 population in 
2013 - were not tested in 2013 because 
they were still in grade 3. Dropping such 
a high proportion of weak students sub- 
stantially inflates the overall gain. 

Further evidence that retention due to 
IREAD-3 rather than other state-level 
policies accounts for the unusually high 
gains in grade 4 comes from the grade 
8 data. The accountability measures and 
new curriculum standards instituted in 
Indiana in the last few years are consis- 
tent across grade level. Thus, if those 
measures were responsible for the grade 
4 gains, there would be similar gains at 
grade 8. The grade 8 mathematics score 
increased more than normal between 
2011 and 2013 but the 2011 score was 
lower than 2009 (Figure 1) suggesting 
that the 2011 score was unusually low 
due to statistical chance. In other words, 
the significant gain in mathematics was 
probably due as much to the surprisingly 
low score in 2011 as it was to gain from 
2011 to 2013. Looking at mathematics 
and reading together (Figures 1 and 3), 
we see that from 2000 to 2013 Indiana 
eighth graders have always been a few 
points above the national average. The 
2013 data fit that pattern. In short, reten- 
tion due to IREAD-3 and normal vari- 
ability in state-level scores, rather than 
state level standards and accountability 
policies, are the major reasons for the 
perceived gains by Indiana students be- 
tween 2011 and 2013 

With respect to achievement gaps, it is 
clear that gaps based on race, ethnicity, 
and SES are substantial and difficult to 
close. We did not report data on gaps 
over time, but when the large margin of 
error in state samples for demographic 
subgroups is taken into account, it does 
not appear that Indiana is doing any bet- 
ter or worse than the nation as a whole 
when it comes to closing gaps. Gender 
gaps are more of an issue in reading than 
they are in mathematics but to the extent 
that there are gaps, they have been rela- 
tively stable. 
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CONCLUSION: TRENDS 
CONTINUE 

Despite claims that the jump in Indi- 
ana’s 2013 NAEP results were due to 
state-level policy changes, the data show 
that the gains are not that much different 
from those of past years. In mathematics, 
Indiana continued its slow, steady im- 
provement, in both average scale scores 
and the percent of students in higher 
achievement levels. Looking at the data 
by gender, race-ethnicity, and free- or re- 
duced-lunch status shows a similar pat- 
tern. In reading, the trend continued to 
be relatively flat in average scale scores 
overall and for subgroups, and there has 
been only modest change in achieve- 
ment levels. Rather than recent changes 
in state policy, it is likely that teachers, 
parents, better curricula, and higher stan- 
dards - factors that have been important 
for a long time - are the drivers of these 
trends. 
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